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INTRODUCTION 


A ssessment and evaluation practices within honors programs have 
attracted considerable attention within the honors academic community, 
e.g., the spring/summer 2006 volume of the Journal of the National Colle¬ 
giate Honors Council. Calls for carefully created and constructed assessment 
activities within honors programs have met with mixed responses by direc¬ 
tors who identify the difficulty in assessing decentralized, complex learning 
environments, noting that standard measures such as tests, surveys, or essays 
are not always applicable or appropriate in addressing honors assessment 
needs, especially in areas of social justice, service learning, and community 
engagement (Corley & Zubizarreta; Lanier). Acknowledging the hesitancy 
of honors directors about the need for assessment as well as their concern 
about the development of authentic assessment practices, Lanier nevertheless 
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encourages honors directors to embrace quality assessment activities as a way 
to demonstrate the value and importance of honors and its enhanced student 
learning. Lanier offers the following comments: 

We now need to do the right thing in honors education and develop 
reliable assessment practices that will generate reliable data and dem¬ 
onstrate convincingly that honors does have the impact on students 
that we all assert as a matter of faith. “Trust me, honors is important 
and our students do very well” just doesn’t work anymore no matter 
how much we may want to fuss or drag our heels. (88-89) 

The focus of this paper is to share one effective method for gathering evidence 
that indicates whether students advanced in their ability to think at a more 
complex level within a short-term honors service learning course. Grounded 
in Wolcott and Lynch’s Steps to Better Thinking Skills model, this augmented 
assessment tool identifies growth in critical thinking skills and areas of intel¬ 
lectual risk. 

ASSESSMENT IN HONORS 

Assessment has become an important component of program develop¬ 
ment and continuation in higher education. Legislators, alumni, parents, 
and students all demand accountability for the learning and skills developed 
in programs offered at universities and colleges. Honors programs are no 
exception. Toward that end, honors directors have increasingly been asked 
to offer evidence supporting the claim that honors programs have value, 
enhance student learning, and provide opportunities for personal growth 
and development. Creating assessment practices, however, can be difficult, 
and developing practices that faculty members can easily integrate into hon¬ 
ors courses is even more of a challenge. As a result, many honors programs 
and colleges have struggled to provide evidence of the value that honors adds 
to students’ educational and personal development, and this is especially 
true for the growing number of programs that emphasize “high-impact edu¬ 
cational practices” (HIP) such as study abroad and service learning (Kuh). 
Klos, Eskine, and Pashkevich note that “questions of social justice and civic 
engagement are an increasing focus of attention in honors education” (53), 
with honors programs offering more activities such as service learning, 
immersion experiences, and community-engaged research. Such experiential 
opportunities play an important role in developing students’ understand¬ 
ing of complex social issues in a global, national, and local setting as well as 
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developing students’ ability to critically examine their personal assumptions 
as well as societal structures. However, assessing this kind of personal and 
academic growth in deep-immersion/high-impact programs is difficult. 

In a 2009 survey of the assessment practices of 24 NCHC members and 
14 non-members, Driscoll found that just over half of the honors programs 
conducted some sort of assessment. Reasons for not engaging in assessment 
practices included “newness of the program, newness of the administrator, 
insufficient time, philosophical opposition to assessment, and an assessment 
plan in process but not in place” (94). In those programs that have engaged in 
assessment, the data collected most often involved student satisfaction with 
honors courses, student satisfaction with the honors program, attrition rates 
from honors programs, causes of honors attrition, and attrition rates of the 
institution. For example, in a longitudinal study of 172 honors students from 
2000 to 2004, Shushok found that honors students had a higher GPA than 
non-honors students at the end of the first year of college, that honors stu¬ 
dents had higher retention rates progressing into the sophomore year, and 
that honors students were more likely than non-honors students to meet 
with a faculty member during office hours, discuss career plans with a fac¬ 
ulty member, or discuss political/social issues with a student outside of class. 
Cosgrove described a similar assessment program, examining the academic 
performance, retention, and degree completion rates of three groups: l) 
honors students who completed the program; 2) honors students who did 
not complete the program; and 3) non-honors students who had similar 
pre-college scores, high school GPAs, and ACT scores as the honors college 
students. Results indicated that the honors students who completed the pro¬ 
gram earned higher GPAs, had higher graduation rates, and graduated in a 
shorter time period than honors students who did not complete the program 
and non-honors students. 

While assessment programs that address the value of honors in relation to 
retention, time to graduation, and enhanced GPA are valuable, less data have 
been collected connecting program assessment to specific learning outcomes. 
According to Driscoll, “Course content and critical thinking were reportedly 
assessed by only 35% of the programs that conduct some assessment (18% 
of the entire sample)” (100). Driscoll’s findings are consistent with previous 
literature reported by Seifert et al. indicating that "relatively little research has 
examined the extent to which honors program participation influences stu¬ 
dent learning" (58). Clearly, honors programs need assessment practices that 
address learning outcomes such as critical thinking skills. 
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EFFECTIVE HONORS STUDENT LEARNING OUTCOMES (SLOS) 

According to Lanier, the first question of good assessment is “What 
do we want our students to learn?” The second is “How do we know they 
learned it?” (90). Toward that end, Zubizarreta identified learning outcomes, 
or domains, for honors students on the NCHC listserv in September 2004, 
which are condensed as follows: 

• Read, write, and think critically 

• Employ an effective process to produce clear, persuasive writing 

• Conduct effective research 

• Develop analysis abilities 

• Integrate active learning and be willing to take learning risks 

• Promote interdisciplinary learning 

• Incorporate community and service learning experiences 

• Demonstrate aesthetic sensitivity 

• Participate actively and effectively in large and small groups 

• Assume multiple roles in groups 

• Demonstrate responsibility outside the classroom and school 

• Demonstrate cultural awareness and gender sensitivity 

• Appreciate learning for its own sake 

• Appreciate diversity 

• Promote effective communication skills 

• Demonstrate personal integrity 

• Develop professional behavior/skills 

• Develop leadership abilities 

• Build moral values/integrity 

• Promote project management and problem solving skills 

• Promote active citizenship roles 
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• Incorporate international experiences 

• Develop foreign language proficiency and 

• Develop creative abilities 

Lanier agreed with Zubizarreta’s listing and added some additional possible 
domains that could be useful in honors assessment: 

• Content (knowledge specific to a discipline or major as well as know¬ 
ledge specific to interdisciplinary or cross-disciplinary activities) 

• Communication (writing skills, oral communication skills, media/ 
computer communication skills, numeric skills, etc.) 

• Critical Thinking 

• Analysis 

• Project management (both group and individual work) 

• Moral values/integrity 

• Problem solving 

• Citizenship 

• Leadership 

• Diversity 

• Creative ability 

• Professional behavior/skills 

• International experience 

• Foreign language proficiency 

• Active learning 

• Interdisciplinary learning 

• Service learning 

• Community service 

• Cultural awareness (90-91) 
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According to Lanier, the first step in the development of an honors assess¬ 
ment plan would be to consider which domains engage honors students in 
specific learning activities that are also central to the mission of the particu¬ 
lar honors program. Lanier focused on the need for, and ability to develop, 
objective questions for assessing these student learning outcomes (SLOs) 
that include the following: 

1. Do our honors programs and colleges actually provide educational 
opportunities and curricular structures that enhance our student’s 
ability to attain these outcomes and goals? 

2. What is the evidence that shows that our honors students have actu¬ 
ally achieved these outcomes? 

Beyond those two fundamental questions are matters of method and prac¬ 
tice: How can an honors program consistently measure the outcomes such 
as “thinks critically” or “achieves strong analytic skills” given the breadth of 
a typical honors program (which is often quite unlike the sharp focus and 
coherence of the curriculum in a major)? What exactly do we mean by these 
outcomes? Where in the honors curriculum do honors students demonstrate 
these behaviors for faculty to gauge? The answers to these questions can pro¬ 
vide evidence of honors students’ academic achievement. The task then is to 
devise specific SLOs that lead to appropriate methods for gathering measur¬ 
able data about whether students are actually learning and accomplishing the 
identified goals. (86) 

CENTRAL MICHIGAN UNIVERSITY HONORS COLLEGE 

The goals and values of a particular honors program play a crucial role 
in the development of its culture (Ford). The Central Michigan University 
(CMU) Honors Program is an active community of scholars that has been 
a campus organization since 1961 and has developed some clearly defined 
goals that include fostering diversity, commitment to academic excellence, 
intellectual engagement, and social responsibility. The program’s mission 
statement emphasizes the need to provide high-ability students with unique 
educational opportunities and experiences; it challenges students to set high 
standards and to achieve academically, personally, and professionally for the 
greater good of our disciplines, our society, and our world. The primary values 
guiding the implementation of this honors program’s mission include criti¬ 
cal thought, scholarly inquiry, creative expression, respect and appreciation 
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for diverse people and ideas in a global society, high standards for integrity 
and personal aspirations, and active citizenship and service for the greater 
good. Students are encouraged to engage with faculty, staff, and fellow stu¬ 
dents through disciplinary and interdisciplinary coursework, seminars, 
philanthropic events, community engagements, and social activities (Central 
Michigan University Honors Webpage). 

All of these criteria help to distinguish the honors student from the non¬ 
honors student at CMU. Honors is not a curriculum, a specific discipline, or 
a program of study. Honors courses are cross-disciplinary experiential learn¬ 
ing contexts designed to connect academic study to broader, more complex 
environments such as service learning, study abroad, or similar events with 
the goal of fostering high-order critical thinking skills and promoting trans¬ 
formative learning. 

COMPLEX THINKING ASSESSMENT INSTRUMENT 

Wolcott and Lynch adapted King and Kitchner’s seven developmental 
stages into five broad patterns of thinking, called “Steps for Better Thinking 
Skill Patterns,” within which people form understandings and beliefs. Level 
0 represents pre-reflective thinking or unexamined assumptions. In Level 1, 
thinkers acknowledge the existence of multiple perspectives but fail to dis¬ 
tinguish between evidence and personal opinion so that reasoning primarily 
consists of gathering information to support existing beliefs. Level 2 thinkers 
can define the problem, identify personal biases, and evaluate multiple per¬ 
spectives, but they lack clarity in defending a particular solution. Level 3 people 
can formulate conclusions by comparing possible alternatives, but solutions 
generally lack meaningful connections beyond the immediate argument and 
fail to include implications and limitations. At level 4 > complex thinkers for¬ 
mulate and contemplate viable solutions with well-grounded arguments and 
an awareness of implications and limitations over time. 

To assist us in charting the levels of complex thinking in student writing, 
we adapted Wolcott and Lynch’s thinking skill pattern descriptors to identify 
nuanced qualities of reasoning. As part of this process, we further expanded 
each level to qualify whether the participant demonstrated (a) weak or incon¬ 
sistent reasoning skills; (b) pervasive, competent, or proficient thought; or 
(c) an awareness, readiness, or attempt to reason at a higher level of thinking 
as shown in Table 1. 
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Our research focused on the nature and evidence of transformative 
learning, especially critical thinking skills. The complexity of arguments from 
student writing could easily be placed within the clear affordances and limita¬ 
tions of thinking described in the ascending levels. 

implementation/assessment process 

We tested our adapted instrument by examining multiple reflective 
writings generated over the duration of an honors service learning course. 
Specifically the goal was to ascertain shifts in thinking, subtle changes in per¬ 
spective, a strengthening of process, or a broadening of vision emerging in 
later reflections as indicators of growth. 

Sixteen traditional honors students were enrolled in a service learning 
course that required them to read and write responses to articles on the dynam¬ 
ics of service and civic engagement; do research and present topics of social 
concern; participate in daily group reflections; and organize and lead a series 
of community service events and projects. Students partnered with agencies 
that addressed issues of environmentalism, cultural preservation, rural and 
alternative education, and elder care. In addition to service activities, students 
completed a series of writing assignments, e.g., a pre-course paper, daily jour¬ 
nals that connected their assigned readings to their service experiences, and 
a final reflection paper due approximately two weeks after the conclusion of 
the experience. As a way to begin the course and to start integrating reflec¬ 
tion and knowledge into the service experience, the first writing assignment 
required students to think about the learning expectations they had for the 
course as well as their interest in and knowledge of an issue specific to the 
community. When discussing the issue, students were cautioned to note the 
complexity of the issue and, if appropriate, various perspectives on it as well 
as to identify possible courses of action to address it. 

Writing assignments required students to reflect on the service experi¬ 
ence, readings, interactions with community members, and their classmates. 
Reflections consisted of two parts: (l) the response that was guided by the 
question posed by the instructors and (2) the student’s personal thoughts 
that might or might not be prompted by the instructors. The reflection pro¬ 
vided a venue for personal synthesis as students interpreted their experiences 
through readings and daily discussions about entering and exiting a commu¬ 
nity, insider/outsider perspectives, and leadership development. In addition 
to the traditional “What?” and “Now What?” and “So What?” questions, the 
prompts addressed situations such as the following: 
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• Why is it important to enter a community effectively? What steps are 
we taking to ensure that we are respectful of this community? 

• How does your insider/outside status impact how you engage in ser¬ 
vice with the island community? 

• How does your insider/outsider status affect how you engage in ser¬ 
vice with this community? 

• What struggles did you encounter and how did you address them? 

A final reflection paper that encouraged students to integrate course read¬ 
ings, service experiences, and future civic engagement activities was due two 
weeks after the conclusion of the course. In the summative writing assign¬ 
ment, students were asked to respond to the following prompts: 

1. Utilize at least 7 of the readings assigned throughout the course, then 
reflect upon what constitutes effective and meaningful service. What 
are the assumptions upon which you have constructed your definition 
of effective service? What elements/ideas need to be incorporated 
into your definition of effective/meaningful service? Please integrate 
the articles carefully into the final paper. 

2. Choose one issue that our service projects have addressed. Think 
about what you initially thought about this issue, what you currently 
think about the issue, and what you have learned about the issue. Inte¬ 
grate the role that the service projects played in your understanding of 
this issue. 

3. Finally, explore what it means personally to be an active citizen and 
the lessons you’ve learned about service and yourself as a result of 
your experiences on the island. Be specific; offer a careful analysis of 
the ideas you offer. What service role will you play when you return 
home? 

In assessing student reflective writings, we bracketed out common termi¬ 
nology from the course readings and familiar phrases from group discussions. 
Instead of knowledge of course content, we looked at how students thought 
about these concepts and experiences as evidenced in opinions, explanations, 
justifications, and other arguments written during the week. Participant 
reflective writings identified four aspects of the course as disorienting: stu¬ 
dent perceptions of community members, the unfamiliar community culture, 
personal leadership roles, and the meaningfulness of service, which Cress et 
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al. describe as disequilibria common to service learning experiences. We high¬ 
lighted unique observations, incongruences, contradictory remarks, and shifts 
in understanding (Daloz 1999), especially when framed within imaginative, 
intuitive, or exceptionally emotive responses to their experiences (Cranton; 
Dirkx; Mezirow). Because we were investigating evidence of growth in criti¬ 
cal thinking skills, we examined how students cognitively processed these 
sites of dissonant experience. 

Although we were aware of comprehension of course content as evidence 
of learning, the research focused instead on how students arrived at their 
conclusions in their reflections. We found evidence of partial or incremental 
steps toward transformative understandings in which students reinforced or 
rethought many of their assumptions during the course. By comparing stu¬ 
dent responses to experienced disequilibria with descriptions in the complex 
thinking assessment model, we were able to locate the students’ range of criti¬ 
cal thinking within specific parameters—even in brief reflective statements. 

STUDENT EXAMPLES 

Although all students and course instructors actively participated in 
the service learning project and advanced in their understanding of course 
content, not everyone was successful in critically assessing new, disquieting 
experiences in a way that was transformative. Four of the participants were 
unable to process the different points of view and continued to dismiss, dis¬ 
count, or ignore what they did not understand. Arguments in their reflections 
offered illogical evidence, expressions of confusion and futility, inappropriate 
application of information from course readings, and reassertions of personal 
experience and opinions as evidence, as described in CTAI Level 0. 

For example, most of the service projects consisted of working with 
elderly community members, and misperceptions of the age group domi¬ 
nated the bulk of initial reflective writings. Most participants contended with 
their misperceptions of the elderly population’s interests, needs, capabilities, 
and values. In their writings, students rethought their original assumptions, 
which ranged from the lack of physical acumen, life experience, knowledge, 
and self-sufficiency to preconceptions about being stubborn, unenthusiastic, 
lonely, senile, and narrow-minded. Participant II’s initial response, however, 
quoted a course reading—“The outsider clearly does not have the direct expe¬ 
rience with the everyday conditions and oppression faced by the community 
members” (Staples 28) [inappropriately applied quote from text to express 
futility of understanding (CTAI 0a)]—and went on to state: 
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The lack of understanding of personal issues a community faces can 
sometimes lead to ignorance among outsiders, and this is a massive 
problem that can lead to ineffective service. Although not always the 
case, actually going through a problem helps a person to become 
truly passionate about a specific issue. This is saying, for example, 
that someone who lived in poverty when he/she was younger is 
more likely to be very dedicated to fighting and ending poverty than 
someone who has been well-off their whole life [illogical argument 
(CTAI Oa)]_Not only do us student volunteers have a good repu¬ 

tation on the island because of past classes and the work they have 
done, but we were also very organized and dedicated throughout the 
week. We were respectful volunteers, doing whatever was asked of 
us, and were very hard workers. We had a solid understanding of the 
issues we were working with, and didn’t necessarily need to adjust 
as much as might be needed in other areas [justification of personal 
behavior; discounting of need to understand elderly (CTAI Oa) ]. We 
all spoke the same language, we were all from the same state, and our 
cultures weren’t very different from one another. There’s not much 
we needed to do that we didn’t do [failure to “acknowledge existence 
of enduring uncertainties and multiple perspectives” (Lynch, Wol¬ 
cott, & Huber) that would indicate logic at Level CTAI Oa]. 

Participant II indicated little or no growth in critical thinking here or in sub¬ 
sequent reflections. 

Other participants demonstrated a shift or elevation in their thinking 
as the course proceeded. Most, when confronted with disequilibria in their 
service-learning experience, were able to identify personal stereotypes or 
unfounded assumptions and made an effort to control biases and evaluate 
evidence from a different perspective. We were able to locate their initial writ¬ 
ings within the descriptors of CTAI Levels 1 and 2. Writings by Participant 
I exemplify how the CTAI evidenced a dramatic shift in a student’s critical 
thinking skills by the end of the course. Reflecting on experiences with the 
elderly in her final paper, she stated: 

Hearing the stories and rich history the elderly told us really changed 
my original stereotype that almost all elderly adults are senile and 
have some sort of dementia. In fact, I even got to hear somewhat of 
the love story regarding how Joe and Lois found each other at low 
moments in their lives. My previous volunteer work with the elderly 
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was in a dementia center, so I had assumed that almost all of the elderly 
would be that way eventually [a previous unexamined assumption; 
“failure to realistically perceive uncertainties/ambiguities” (CTAI 
Oa)]. I was practically ashamed of myself for having thought such 
things because these individuals had such an incredible amount of 
experience with their lives, and one of the things that I took away 
from this was to understand that these people had so much to offer 
[two arguments from personal observation; “reaches own conclu¬ 
sion ...” (CTAI lb)]. Therefore, it’s clear that I have learned a great 
deal about this issue. More specifically, I’ve learned that ageism is just 
as bad, if not worse, than racism or sexism. It can make people who 
are elderly feel useless or incompetent, which is the opposite of what 
they should be feeling since they have attained the accomplishment 
of making it so far in life [“presents coherent and balanced descrip¬ 
tion of a problem and the larger context in which it is found” (CTAI 
2a)]. I’ve also learned the classic lesson of never judging a book by 
its cover. It’s easy to look at someone and make so many assumptions 
about them, but this is incredibly unfair because that person is never 
given the chance to explain their story or show what they are capable 
of [identifies issue and cause of bias (looks), and qualitatively evalu¬ 
ates the unfairness (CTAI 2c)]. 

However, most participants experienced moderate shifts in their thinking, 
and the complex thinking assessment instrument was able to identify gradual 
assimilation of understanding or shifts in thinking or changes in perspective. 
Participant XIII is an example of this more subtle growth. At the beginning of 
the course, her reflective response to interacting with the elderly began with a 
guarded perspective but articulated why she might not understand: 

[B]eing a younger and physically more able person, it is hard for me 
to not only understand the community’s perspectives, since I have 
grown up in a different time and not lived nearly as long, but grasp 
obstacles associated with age, since I have not yet experienced it 
[“expresses confusion” and “cannot evaluate or appropriately apply 
evidence” (CTAI Ob)]. I have an advantage in this area, because I 
have at least some knowledge on most of the issues that we have been 
working to solve, which include the environment, health, and care 
for the elderly. 
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In her final paper, after further work with the elderly during the course and 
participating in group discussion, Participant XIII rethought these issues in 
relation to prior experience and how she might engage others in the future: 

We also noted that we don’t go and help our grandparents with yard 
work very often either. This whole experience and reflection as a 
group taught me how important it is to keep an open mind, because 
you never know how much someone might need the help. I think 
this will transfer over into my future service, because I will be more 
willing to keep an open mind of new activities and listen to whatever 
the person I’m working with needs done [“reaches own conclusion 
without relying exclusively on authority;” defines problem and shifts 
argument based on evidence from “personally relevant aspects of 
problem” (CTAI lc)]. 

These small, evolutionary statements acknowledge the existence of multiple 
perspectives, identify personal assumptions and control biases, and reach per¬ 
sonal conclusions that represent a subtle elevated change in Participant XIII’s 
critical thinking skills. By changing what she thought and how she arrived at 
those conclusions, she demonstrated an awareness and willingness to think at 
a more complex level (CTAI 2a). 

By reading for argument rather than content and comparing the qual¬ 
ity of thinking to CTAI descriptors, we could identify positive and negative 
attributes at each level that became clear parameters within which to place 
patterns of thinking. Pre-assessment practice and discussion clarified the pro¬ 
cess, and frequent double-checking between raters maintained consistency 
and inter-rater reliability. 

When using this instrument, it is important to notice that few adults reach 
levels 3 and 4 without college training, that there should be little expectation 
to witness dramatic transition within the confines of a single course, and that 
the four ascending levels of reflective thinking develop over an adult’s lifetime 
(Dirkx; Wolcott & Lynch). 

IMPLICATIONS 

The use of the Complex Thinking Assessment Instrument can allow 
an honors program to shift the focus of assessment away from solely reten¬ 
tion percentages, grade point averages, numbers of individuals participating 
in service experiences, and graduation rates, to critical thinking skills and 
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student learning outcomes, thus providing a better mechanism for describing 
growth in complex thinking as well as understanding in the context of hon¬ 
ors. The development of a qualitative assessment instrument that identifies 
qualities and levels of complex thinking can document that Student Learn¬ 
ing Objectives are met within courses and within the honors program as a 
whole by showing growth in complex thinking skills. Students benefit from 
this assessment because they are more likely to develop their thinking skills 
if they understand the goal(s) and receive explicit feedback on their perfor¬ 
mance (Lynch & Wolcott). The use of this tool has benefits for faculty as well 
as students. The challenge facing honors faculty members is how to acquire 
the ability to produce desirable honors educational practices, construct 
knowledge relative to the distinctive nature of honors education, and use 
reliable, verifiable assessment practices to enhance honors pedagogy, hon¬ 
ors curricular and instructional design, and honors educational experiences 
to generate reliable data and credibly demonstrate that honors does have an 
impact on students (88-89). In order to assure that faculty get the training 
and resources necessary to improve their ability to do this kind of assessment, 
institutional support and commitment are needed (Molee). 

The honors curriculum is an effective place to promote enhanced critical 
thinking, and faculty members play a crucial role in guiding students toward 
increasingly complex thinking. The willingness of honors faculty to engage in 
a dialogue that increases their ways of knowing enhances their understanding 
of these challenges, informs curriculum development, increases understand¬ 
ing of what makes the honors student unique, and enhances a successful 
honors experience. 

Lanier extended an appeal for help in developing assessment tools that 
show the gains by honors students (1) as compared to their non-honors coun¬ 
terparts and (2) as a result of their shared educational enrichment practices. 
Use of this tool can assist in providing evidence that critical thinking skills 
develop not just in one course but throughout the honors students’ university 
experience; it can be used to measure the changes in levels of complex think¬ 
ing from enrollment in college to graduation. Used properly, the tool can (l) 
demonstrate the development of higher-order thinking skills among honors 
students over their entire academic experience, 2) indicate gains that honors 
students make in comparison to their non-honors peers, and 3) document the 
success of enrichment practices—i.e., cultural trips, international education, 
campus leadership, citizenship, active learning experiences, service learning, 
and community service—that characterize successful honors programs. 
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